Do Decision-Analytic Models Identify Cost-Effective Treatments? A Retrospective Look at Helicobacter Pylori Eradication

BACKGROUND: Pharmacoeconomic models of Helicobacter (H) pylori eradication have been frequently cited but never validated. OBJECTIVES: Examine retrospectively whether H pylori pharmacoeconomic models direct decision makers to cost-effective therapeutic choices. METHODS: We first replicated and then validated 2 models, replacing model assumptions with empirical data from a multipayer claims database. Database subjects were 435 commercially insured U.S. patients treated with bismuthme tronidazole-tetracycline (BMT), proton pump inhibitor (PPI)-clarithromycin, or PPI-amoxicillin. Patients met greater than1 clinical requirement (ulcer disease, gastritis/duodenitis, stomach function disorder, abdominal pain, H pylori infection, endoscopy, or H pylori assay). Sensitivity analyses included only patients with ulcer diagnosis or gastrointestinal specialist care. Outcome measures were: (1) rates of eradication retreatment; (2) use of office visits, hospitalizations, endoscopies, and antisecretory medication; and (3) cost per effectively treated (nonretreated) patient. RESULTS: Model results overstated the cost-effectiveness of PPI-clarithromycin and underestimated the cost-effectiveness of BMT. Prior to empirical adjustment, costs per effectively treated patient were $1,001, $980, and $1,730 for BMT, PPI clarithromycin, and PPI-amoxicillin, respectively. Estimates after adjustment were $852 for BMT, $1,118 for PPI-clarithromycin, and $1,131 for PPI-amoxicillin. Key model assumptions that proved retrospectively incorrect were largely unsupported by either empirical evidence or systematic assessment of expert opinion. CONCLUSIONS: Organizations with access to medical and pharmacy claims databases should test key assumptions of influential models to determine their validity. Journal peer-review processes should pay particular attention to the basis of model assumptions.

of service costs but on findings of decision-analytic models that had begun appearing in the literature in 1995. [25][26][27][28][29][30] Two key features of these models explained these conclusions. First, models comparing outcomes for different eradication regimens assumed that BMT' s effectiveness in clinical practice would be compromised by noncompliance and metronidazole resistance. 29,30 Second, models assumed that costs associated with disease recurrence (e.g., office visits, laboratory work, and hospitalizations) would far exceed the costs of the regimen drugs themselves, making the recurrence rate "the principal determinant of overall cost" 21 in model results. Recurrence-related costs, it was argued, were so high that medication costs were "minor" 22 to economic outcome.
A more recent study was the first to measure empirically the retreatment rates and medical service costs for patients treated with different eradication regimens in routine clinical practice. 31 Controlling for patient age, gender, use of gastrointestinal (GI) specialty care, and clinical criteria, retreatment rates were lower for 3-drug than 2-drug therapies and not significantly different for BMT versus PPI 3-drug therapies. 31 Consistent with the view that recurrence resulted in additional medical expense, retreated patients incurred higher medical costs than nonretreated patients. However, in most medical service categories, follow-up costs for patients taking less effective versus more effective regimens were not significantly different. 31 These findings raised questions about the degree to which decision analytic models of H pylori eradication had directed decision makers to cost-effective therapeutic choices.
The present study explored that issue, responding to the call to validate and update models using "real life" outcomes data. 3,32 The 2 models examined in this study, Model A 30,33 and Model B, 29 were selected because they are widely cited, well documented, and the only models to compare outcomes for patients treated with different H pylori eradication regimens in the United States. We first replicated the models' results. Then, in place of model assumptions, we substituted actual service utilization measures derived from a database of integrated medical and pharmacy claims.
Research questions included: (1) To what extent do modelcalculated and actual retreatment rates match? (2) To what extent do models' assumed recurrence-related services match actual medical service-use patterns? (3) When models are empirically adjusted, do conclusions about the cost-effectiveness of different therapeutic choices change?
ss Methods Models were replicated using Data 3.5 for Health Care (TreeAge Software). 34 Actual service utilization information came from an integrated medical and pharmacy claims database of commercially insured patients enrolled in several managed care organizations (approximately 70% independent practice association [IPA] and 30% group model) located throughout the United States. The 435 patients included in the present analysis are a subset of a sample used in a previously published study of retreatment rates and medical costs for patients aged >16 years who were treated with H pylori eradication regimens. 31 The present study' s patients were treated with one of 3 regimens common to Models A and B: BMT (N = 98), PPI-clarithromycin (N = 207), and PPI-amoxicillin (N = 130). Of BMT-treated patients, 24% filled a prescription for an H2 receptor antagonist (H2RA) along with the eradication regimen medications.
Although Model A did not initially include patients treated with PPI 3-drug regimens, final analyses in the present study included an additional 119 patients treated with the 3-drug combination PPI-clarithromycin-metronidazole. These analyses were performed for 2 reasons. First, PPI-based 3-drug regimens are currently recommended, while PPI-based 2-drug regimens are generally recognized today as ineffective. 35 Second, after its initial development, Model A was used to assess the cost-effec- tiveness of PPI-based 3-drug regimens. 33 The database' s and models' study time periods were approximately the same. Database patients were continuously eligible from April 1, 1995, through December 31, 1996, and filled prescriptions for an H pylori eradication regimen during the time period June 1, 1995, through May 31, 1996. Each patient' s follow-up period began on the date that the regimen prescriptions were filled and ended on the study end date, December 31, 1996.

Assumptions Used in Original Models of Helicobacter Pylori Eradication Rates
Database patients met one or more of the following clinical criteria during the 60 days + the regimen date: (1) primary, secondary, or tertiary diagnosis (International Classification of Diseases, 9th revision, [ICD-9]) code for ulcer disease (531XX-534XX), gastritis/duodenitis (535XX), stomach function disorder (536XX), abdominal pain (7890X), or H pylori infection (041.86), or (2) Current Procedural Terminology (CPT) code for endoscopy (432XX) or H pylori assay (86677). To reflect typical clinical practice, all patients meeting these criteria were included regardless of dose or duration of eradication treatment.
Database diagnostic criteria were consistent with those of the models, both of which have been applied to populations treated for H pylori infection irrespective of whether the precipitating diagnosis was ulcer disease or another condition (e.g. gastritis or nonulcer dyspepsia). 29,33 However, because Model A's patient population initially was limited to duodenal ulcer patients with confirmed infection, 30 we performed sensitivity analyses on a subset of patients with a diagnosis of ulcer disease combined with either diagnosed H pylori infection or procedure codes for endoscopy or H pylori assay. Additional sensitivity analyses limited the sample to patients with >1 claim from a GI specialist during the time period from 30 days before through 7 days after the initial regimen date because GI specialists are more likely than primary care physicians to treat H pylori infection appropriately. 36

Comparison of Models-Calculated With Actual Retreatment Rates
To replicate Model A's recurrence rates, baseline eradication rates for each regimen were reduced by factors representing assumed noncompliance and, for BMT patients only, assumed metronidazole resistance. The resulting eradication rates were then weighted by assumed recurrence rates for patients with (2%) and without (86%) bacterial eradication. 30 For Model B patients, baseline eradication rates for each regimen were reduced by assumed noncompliance factors to replicate that model' s compliance-adjusted result. 29 For database patients, retreatment was defined as filling a second set of eradication regimen prescriptions (for either the same or a different regimen) during the follow-up period. For both database and model patients, "best case" and "worst case" retreatment rates were calculated. For database patients, the best-case and worst-case rates were, respectively, the lower and upper limits of the 95% confidence intervals around actual retreatment rates. Calculation of best-case and worst-case rates for the models were based on model assumptions and sensitiv-ity ranges. Best-case rates used the lower limits of models' sensitivity ranges for metronidazole resistance and recurrence and the upper limits of sensitivity ranges for compliance and eradication rate (Table 1). Worst-case rates used the opposite ends of the sensitivity ranges.
To address the possibility that some patients with recurrent symptoms were treated with antisecretory therapy instead of repeat eradication therapy, a sensitivity analysis defined a patient as having persistent symptoms, albeit not necessarily recurrent infection, if either retreatment or initiation of antisecretory therapy occurred during the follow-up period. This approach might seem counterintuitive since the NIH consensus statement recommended eradication therapy on either initial or subsequent disease presentation. 9 We made the calculation because Model A assumed that all patients with recurrent disease would be treated with antisecretory therapy alone and that patients without recurrent disease would receive no antisecretory drugs.
Our original analysis plan was to substitute actual retreatment rates into both models and assess the impact on modelcalculated total medical cost. However, we were able to replicate costs fully for Model A only. After consultation with one of Model B' s authors, further attempts to replicate costs for Model B became infeasible due to author availability problems. Thus, only retreatment rates are presented for Model B.

Comparison of Models-Assumed With Actual Medical Services
Model A assumed that all patients would receive initial and final office visits plus the H pylori eradication medications and that patients with recurrent disease would receive additional medical and pharmacy services attributable to disease recurrence. We updated Model A with database-derived utilization rates for both retreated and nonretreated patients. (Methodological note: In calculating cost attributable to retreatment, this approach is mathematically equivalent to updating the model with the difference between the utilization rates for retreated and nonretreated patients. For example, if a service is used by 50% of retreated and 30% of nonretreated patients, we could either directly substitute those figures into the model or assume a retreatment-related utilization rate of 20% (with 0% assumed for nonretreated patients). We selected the former approach because it is more straightforward and reflects actual service use.) For each retreated database patient, a medical service (office visit, endoscopy, or hospitalization) was defined as retreatmentrelated if it occurred during the 45-day period from 14 days prior through 30 days after (-14/+30) that patient' s retreatment date. To avoid confusing retreatment-related costs with initial treatment costs, we eliminated from the medical service analyses 2 patients whose retreatment occurred fewer than 15 days after the initial treatment. To have 30 days of postretreatment data for all patients, we also eliminated 7 patients whose retreatment occurred fewer than 30 days prior to the end of the study. Thus, of 58 retreated patients, 49 were included in these analyses. To obtain comparable data for nonretreated patients, we first calculated the mean number of days from initial treatment to retreatment for retreated patients. That figure-117 days-was added to each nonretreated patient' s initial eradication regimen date to produce the nonretreatment-comparison date. The time period from 14 days prior through 30 days after the nonretreatment-comparison date was then used to calculate nonretreatment-comparison utilization and costs. Sensitivity analyses measured the percentages of retreated and nonretreated patients with any GI service use in the follow-up period, regardless of the timing of the service use relative to the retreatment date.

Model A Baseline Replication
We defined GI costs to include not only ulcer disease but also a wide variety of GI symptoms. Thus, we identified office visit and hospital services as GI-related if they either met the diagnostic or procedural criteria for study inclusion or included any of the following ICD-9 codes: 530XX (diseases of esophagus), 537XX (other gastroduodenal disorder), 787XX (GI system symptoms), 5781X (blood in stool), 564XX (functional digestive disorder), or 5699X (intestinal disorder not otherwise specified). For any hospital stay in which either the primary, secondary, or tertiary diagnosis for >25% of claim lines for either professional or facility services met these diagnostic criteria, costs for all professional and facility claim lines were summed to calculate hospitalization cost.
Although Model A assumed only use of H2RAs in assessing antisecretory medication use, we also included PPI use to be as consistent as possible with the model assumption of high recurrence-related costs. Antisecretory drugs were identified using a Generic Product Identifier (GPI) code beginning with either "4920" (H2RA) or "4927" (PPI). 37 The percentages of patients with ≥1 claim for an antisecretory medication at any time following the retreatment date (retreated patients) or retreatmentcomparison date (nonretreated patients) were calculated. The percentages of retreated and nonretreated patients still using antisecretory medication as of the study end, defined as having ≥1 claim within 45 days of December 31, 1996, were also calculated. Model A assumed that antisecretory use continued for 18 months for all patients with recurrent disease. To be as consistent as possible with model results, we assumed that all patients still using antisecretory medications at the study end date continued to use them for the full 18 months assumed by Model A. For patients not still using antisecretory medications at study end, we calculated the mean length of actual use and substituted that figure into the model.
We substituted actual allowed billed charge data into Model A instead of the model' s assumed costs. Allowed billed charges exclude noncovered services but do not reflect payers' discounts off the total billed charge. Thus, results are unaffected by discounts unique to any particular health plans. To assess expenses for initial and final office visits, we calculated the mean charge for all GI-diagnosis office visits occurring on or up to 7 days prior to the regimen date. Mean charges for office vis-its and endoscopies were calculated for the -14/+30-day period. Mean monthly charges for antisecretory drugs were calculated for all claims after the retreatment date. Mean hospital charges were calculated for the -14/+30-day period. Since only 7 hospitalizations occurred during that time, we also calculated charges for the 70 hospitalizations occurring throughout the follow-up period as a sensitivity analysis. This methodology is similar to Model A's, which used national average inpatient charges for ulcer diagnoses as the source of its hospital cost figure.

Cost Per Effectively Treated Patient
For the original Model A, the effective treatment rate was 1 minus the estimated recurrence rate. For the empirically revised model, the effective treatment rate was 1 minus the actual retreatment rate. For both models, cost per effectively treated patient was calculated as total estimated cost per treated patient divided by the effective treatment rate.

ss Results
Model A's reported outcome-total cost associated with each of the 3 regimens-was replicated to within $1 for 2 of the regimens (BMT and PPI-clarithromycin) and to within $3 for PPIamoxicillin ( Figure 1). Model A's sensitivity analyses were also replicated. Model B' s recurrence rates were replicated exactly for PPI-clarithromycin and PPI-amoxicillin, and within 1 percentage point for BMT.

Retreatment Rates
Both models' results overstated actual retreatment rates ( Table  2). Across all regimens, model-calculated rates were higher than actual rates of retreatment. The discrepancy between modeled and actual retreatment rates was particularly large for BMT. Model A's "Base Case" retreatment rates for BMT, PPI-clarithromycin, and PPI-amoxicillin exceeded actual rates by 367%, 57%, and 122%, respectively, and exceeded the upper limit of actual rates' 95% confidence interval ("Worst Case") by 155%, 22%, and 60%. For the same regimens, Model B' s Base Case exceeded actual rates by 183%, 100%, and 78% and exceeded the upper limit of actual rates' 95% confidence interval by 55%, 56%, and 28% respectively.
Results were similar when the sample was limited to patients with (1) diagnosed ulcer disease in combination with an H pylori diagnosis, endoscopy, or H pylori assay or (2) GI specialty treatment (not shown). 38 When persistent symptoms were defined to include either retreatment or antisecretory therapy, rates were considerably higher for all regimens, as expected, but the pattern of differences among the regimens remained similar.

Patterns of Care for Retreated and Nonretreated Patients
Nonretreated patients generally used more services, and retreated patients used fewer services than assumed in Model A ( Table  3). The model assumed that 100%, 100%, and 50% of patients with recurrent disease would use antisecretory medications, office visits, and endoscopies, respectively. The model further assumed use rates of 0% for these services for nonrecurrent patients. Actual respective use rates for these services were 65%, 41%, and 6% for retreated and 46%, 10%, and 2% for nonretreated patients.

Assumptions About Patterns of Medical Care for Retreated and Nonretreated Patients
Revised per-service charges were somewhat higher than costs originally assumed by Model A for office visits and antisecretory medication but less than half that assumed for endoscopies ( Table 3). The model assumed that all hospitalizations were inpatient at $22,809 each, but more than 80% of the 70 hospital episodes during the follow-up period, and 4 of the 7 hospitalizations during the -14/+30-day period were outpatient, making actual charges substantially lower than assumed charges. Mean (median) charges per hospitalization were $5,022 ($1,488) for the 7 hospitalizations occurring during the -14/+30 period and $3,256 ($1,394) for the 70 hospitalizations occurring during the entire follow-up period. To be as consistent as possible with the model' s assumption of high recurrence-related costs, we used the higher $5,000 figure in the revised model.
Sensitivity analyses of service-use rates produced similar findings, whether measuring service use for the entire study period, for ulcer patients, or for patients treated by GI specialists (not shown). 38 For nearly every service category, utilization rates were significantly higher for retreated than nonretreated patients. However, use rates for retreated patients were not as high as assumed by Model A, and nonretreated patients' utilization rates consistently exceeded 0%.

Effect of Changed Assumptions on Results
Revising Model A to reflect actual retreatment rates and service costs produced substantial changes in results (Table 4, Figure  2), with BMT more cost effective and PPI-clarithromycin less cost effective than originally estimated. In the original replicated Model A, costs per effectively treated patient were slightly higher for BMT and much higher for PPI-amoxicillin than for PPI-clarithromycin. After revising Model A for actual retreatment rates, service utilization, and costs, 2-year costs per effectively treated patient were much higher for either PPI-clarithromycin or PPI-amoxicillin than for BMT.
Results were similar when the model was further updated to include patients initially treated with PPI-clarithromycin-metronidazole (Table 4, Figure 3). Recurrence-associated costs changed, albeit very slightly, because utilization data for retreated PPI-clarithromycin-metronidazole patients were incorporated. After this adjustment, costs per effectively treated patient were $847 for BMT, $982 for PPI-clarithromycin-metronidazole, $1,123 for PPI-amoxicillin, and $1,111 for PPI-clarithromycin.
ss Limitations This is one of few studies to validate pharmacoeconomic decision-analytic models. The findings corroborate the findings of other critical reviews 39 and raise questions about models' role in guiding treatment selection. Before considering these implications, potential limitations should be discussed.
One is the possibility that some nonretreated database patients actually had persistent infections that will result in future recurrences and medical cost. Indeed, if one assumes that 100% of patients with persistent infection seek treatment, database retreatment rates are lower than expected given published efficacy data. 12,31 However, this study meets a need for outcomes research recognizing that patients with persistent H pylori infection are not always sufficiently symptomatic to seek medical attention. 40 Moreover, the follow-up period to identify retreatments in the database study (range 7 to 19, median 12.5 months) was generous compared with model parameters. Under Models A and B, respectively, recurrences were assumed to occur at 6 months and 4 to 7 months.
An additional potential concern is selection bias, i.e., that patients whom physicians knew to be less compliant were prescribed PPI-based regimens because these were considered more tolerable than BMT. This possibility is belied by the similarity between the present study' s results and findings of randomized clinical trials. Specifically, based on the results of randomized clinical trials, one would expect the lowest retreatment rates for BMT and PPI-clarithromycin-metronidazole, higher rates for PPIclarithromycin, and the highest rates for PPI-amoxicillin, 11-13 consistent with what was observed in the claims database. Another potential issue is inaccuracy inherent to use of a claims database, particularly imprecision in diagnostic coding. While this possibility exists, there is no reason to believe it affected patients treated with any particular regimen. Moreover, neither utilization nor retreatment rate findings were sensitive to stringency of diagnostic criteria for study entry or receipt of specialty care.

Changes to Cost-Effectiveness Findings After Modifications to Model A
These findings represent just one database, albeit a geographically diverse and multipayer database. Results for other populations could be different. We urge payers with access to integrated medical and pharmacy data to conduct similar validations of these and other models to test key model assumptions and results against empirical evidence.
A final potential limitation is inherent to all models. We have adjusted a model' s 2-year cost projections based on data from a shorter time period. Even an adjusted and more accurate model is still only a model, not a direct cost measurement over the full 2 years. In particular, the true length of antisecretory therapy is unknown. The assumption that all patients still using antisecretory drugs at study end continued to use them for 18 months is somewhat questionable but was made to be as consistent as possible with Model A' s original assumptions.

ss Conclusions
Findings derived from this study' s multipayer database suggest that the pharmacoeconomic model results for H pylori eradication did not direct decision makers to cost-effective treatment choices. BMT' s retreatment rates were overestimated by both models. Moreover, after adjusting Model A to reflect actual retreatment rates and service-use patterns, BMT was more cost effective and PPI-clarithromycin less cost effective than Model A had originally indicated.
But did the publication of these pharmacoeconomic models actually affect the behavior of decision makers such as physicians and formulary developers? One might argue that, since the limitations of models are widely recognized, the present study' s findings are of little import. Research and policy literature published in the years following the models' development refute that view since models were commonly cited to justify use of more expensive drug regimens. [20][21][22][23][24]41 For example, in a journal targeted to formulary decision makers, a physician panel member citing Model A concluded in 1998 that "if a regimen costs $100 but fails to cure the patient' s disease, then endoscopy will be needed … at a cost of about $1,000." 41 Similarly, a 1998 review article, appearing in a prominent journal targeted to physicians, cited Model A to document that "ineffective treatment regimens increase the costs associated with treatment because they result in the need for continued interaction with health care providers, further procedures, and maintenance therapy. 23 Notably, in both instances, the cited information represented model assumptions, described as if they had been empirically determined.

Model A Empirically Adjusted for Actual Retreatment Rates and Utilization
Consideration of specific problems in the models can guide recommendations to enhance models' usefulness. Several problems in key assumptions affected model results.
First, the assumption that all successfully treated patients discontinue antisecretory medication, although common to H pylori eradication models at that time, [25][26][27][28] appears to have been incorrect. The present study findings are consistent with more recent research indicating use of antisecretory drugs by substantial portions of patients after eradication of H pylori infection. 42 Second, the assumption that all hospital care would be inpatient care also proved erroneous in that the majority of followup hospital episodes were outpatient. Correction of this assumption reduced the estimated cost for patients receiving hospital services from more than $24,000 to approximately $6,000.
Third, assumptions about the degree of noncompliance and its impact on patient outcomes played a substantial role in discrepancies between modeled and database outcomes. Model A 30 assumed compliance of 65% with triple therapy (bismuthmetronidazole-tetracycline) and 95% with PPI-clarithromycin and PPI-amoxicillin regimens even though in most clinical trials published before 1996, dropout rates for all regimens including triple therapy were less than 5%. 43 The highest dropout rate that had been observed for triple therapy was 20% in a nonrandomized trial with 40 patients. 44 Model B' s compliance assumptions were based on a study of NSAID therapy dosing schedules. 29 Whether NSAID and H pylori therapies are sufficiently comparable to generate similar compliance patterns is open to question.
Fourth, Model A's sensitivity testing included primarily 1-factor analyses that did not foresee the discrepancies in multiple assumptions observed in our study. The only reported  2-factor analysis included a sensitivity test on metronidazole resistance and compliance in which compliance was assigned an upper limit of 80% for BMT and a lower limit of 75% for the PPI-based regimens.
The choices made by the authors of these models demonstrate the "gray areas" that decision-analytic modelers face when they try to predict outcomes of new treatment approaches. For example, any compliance assumptions made in 1996 were unavoidably speculative because empirical data on compliance rates with H pylori eradication in clinical practice were not available. However, publications relying on H pylori model results in making cost-effectiveness assessments [20][21][22][23][24] were appearing in the literature at the same time that some physician letters and commentaries were questioning model assumptions. [45][46][47][48] These problems demonstrate the value of standard techniques to ascertain expert opinion, such as a Delphi panel. Neither Model A nor Model B used an expert panel in developing compliance assumptions, although Model B used an expert panel in developing utilization assumptions.
An additional important step would be the increased use of sophisticated modeling techniques to reflect more precisely the probabilities of various treatment outcomes. A recent study applying "bootstrapping" methodologies to Model B suggested that the revised method yielded additional insights into H pylori eradication. 49 Such techniques might be particularly useful when there is truly no way to predict a clinical outcome such as, for example, the rate at which patients will continue to use antisecretory drugs after successful H pylori eradication.
Despite the overall transparency and thorough documentation of these 2 models, replicating them was not an easy task, and we were unable to fully replicate costs for Model B. To facilitate validation efforts of this type, we would echo opinions that electronic versions of models should be made readily available to other researchers after publication and a required part of submission for prepublication peer review. 3,32,50 Our findings raise concerns about peer review of these models. Models are, by definition, heavily reliant on their assumptions. Particularly in Model A, key model assumptions about compliance and recurrence-associated utilization were largely unsupported. Whether models are using the best available information should be a key part of the peer-review process.
One might argue that the need for rapid information about newly introduced drugs is so acute that the use of "common sense," albeit unstudied, assumptions is sometimes necessary, recognizing that no pharmacoeconomic model is ever 100% accurate. In other words, a balance between speed and accuracy will remain an unavoidable fact of life in pharmacoeconomics. However, when inaccuracies in a model' s assumptions are so inconsistent across treatments that empirical correction of the model changes treatment recommendations, one must question the usefulness of the model. This question, and the more general question of what criteria should be used to assess the validity of pharmacoeconomic models, should receive more serious attention.
While errors in economic modeling may not be as troubling as errors in studies of efficacy or safety, preventing and correcting such errors is in patients' best interest. Since health care resources are finite, payer funds expended on one treatment are, by definition, unavailable for others. Moreover, since patients commonly pay portions of both monthly premiums and perprescription costs, promoting cost-effective treatment decisions ultimately controls not only payers' costs but also patients' outof-pocket expenditures.
Our findings reaffirm the oft-expressed need for ongoing examination of the validity of pharmacoeconomic models.